Skip to content

docs(examples): add nemoguards cache configuration example #1459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

docs(examples): add nemoguards cache configuration example #1459

wants to merge 2 commits into from

Conversation

Pouyanpi
Copy link
Collaborator

@Pouyanpi Pouyanpi commented Oct 17, 2025 •
edited
Loading

Description

This is PR 5/5 (final) in the NeMoGuards caching feature stack.

Test Plan

results:

======================================================================
MODEL CACHE CONFIGURATION
======================================================================
✅ content_safety: maxsize=10000, track_stats=True
✅ topic_control: maxsize=10000, track_stats=True
✅ jailbreak_detection: maxsize=10000, track_stats=True

======================================================================
GENERATING 1000 TEST PROMPTS
======================================================================
Generated 1000 prompts (100 unique × 10 repetitions)

======================================================================
PROCESSING PROMPTS
======================================================================
Processed 100/1000 prompts - Hits: 270, Misses: 30, Errors: 0
Processed 200/1000 prompts - Hits: 540, Misses: 60, Errors: 0
Processed 300/1000 prompts - Hits: 810, Misses: 90, Errors: 0
Processed 400/1000 prompts - Hits: 1080, Misses: 120, Errors: 0
Processed 500/1000 prompts - Hits: 1350, Misses: 150, Errors: 0
Processed 600/1000 prompts - Hits: 1620, Misses: 180, Errors: 0
Processed 700/1000 prompts - Hits: 1890, Misses: 210, Errors: 0
Processed 800/1000 prompts - Hits: 2160, Misses: 240, Errors: 0
Processed 900/1000 prompts - Hits: 2430, Misses: 270, Errors: 0
Processed 1000/1000 prompts - Hits: 2700, Misses: 300, Errors: 0

Completed processing 1000 prompts

======================================================================
CACHE STATISTICS FROM LOGS
======================================================================
content_safety       - Hits:  900, Misses:  100, Hit Rate:  90.0%
topic_safety         - Hits:  900, Misses:  100, Hit Rate:  90.0%
jailbreak            - Hits:  900, Misses:  100, Hit Rate:  90.0%

======================================================================
CACHE STATISTICS FROM CACHE OBJECTS
======================================================================
content_safety       - Size: 100/10000, Hits: 900, Misses: 100, Hit Rate: 90.00%
topic_control        - Size: 100/10000, Hits: 900, Misses: 100, Hit Rate: 90.00%
jailbreak_detection  - Size: 100/10000, Hits: 900, Misses: 100, Hit Rate: 90.00%

======================================================================
VALIDATING traces.jsonl
======================================================================
Total LLM spans in traces: 3000
Cached LLM spans (llm.cache.hit=true): 2700

content_safety       - Total: 1000, Cached:  900, Hit Rate:  90.0%
topic_safety         - Total: 1000, Cached:  900, Hit Rate:  90.0%
jailbreak            - Total: 1000, Cached:  900, Hit Rate:  90.0%

Traces file: /Users/prezakhani/git/nemoguardrails-project/nemo-guardrails-review/.traces/trace.jsonl

======================================================================
SUMMARY
======================================================================
✅ Processed 1000 prompts with input rails (content safety, topic safety, jailbreak)
✅ Total cache hits: 2700
✅ Total cache misses: 300
✅ Overall cache hit rate: 90.0%
✅ Traces validated for llm.cache.hit attribute across all safety checks

This was referenced Oct 17, 2025
Merged
Merged
@Pouyanpi Pouyanpi force-pushed the feat/cache-jailbreak-detection branch from 6a7474e to 630eafa Compare October 17, 2025 10:35
@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Pouyanpi Pouyanpi force-pushed the feat/cache-jailbreak-detection branch from 630eafa to 9dbb323 Compare October 17, 2025 10:40
@Pouyanpi Pouyanpi force-pushed the docs/cache-nemoguards-example branch from 9495808 to 27685a2 Compare October 17, 2025 10:41
This was referenced Oct 17, 2025
Merged
Merged
@Pouyanpi Pouyanpi added this to the v0.18.0 milestone Oct 17, 2025
@Pouyanpi Pouyanpi self-assigned this Oct 17, 2025
tgasser-nv
tgasser-nv reviewed Oct 17, 2025
View reviewed changes
Copy link
Collaborator

@tgasser-nv tgasser-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a description of how this config is different to the regular hero-workflow one (i.e. caching) and how they're configured. I can update the PR as you have so many others on your plate

examples/configs/nemoguards_cache/README.md Outdated
@@ -0,0 +1,24 @@
# NeMoGuard Safety Rails Example

This example showcases the use of NVIDIA's NeMoGuard NIMs for comprehensive AI safety including content moderation, topic control, and jailbreak detection.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add some more description about how this enables caching, and how this is different from our regular hero-workflow config (nemoguards)? I'm happy to do this as you have so many other PRs to review

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated the example based on #1436

@Pouyanpi Pouyanpi force-pushed the feat/cache-jailbreak-detection branch from 9dbb323 to d38eb0b Compare October 19, 2025 10:26
Base automatically changed from feat/cache-jailbreak-detection to develop October 19, 2025 10:38
@Pouyanpi Pouyanpi force-pushed the docs/cache-nemoguards-example branch from 9649c8a to 9cbb265 Compare October 20, 2025 10:54
@Pouyanpi Pouyanpi force-pushed the docs/cache-nemoguards-example branch from 9cbb265 to ebb5f09 Compare October 20, 2025 10:55
@github-actions GitHub Actions
Copy link
Contributor

Documentation preview

https://nvidia-nemo.github.io/Guardrails/review/pr-1459

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tgasser-nv tgasser-nv tgasser-nv left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@Pouyanpi Pouyanpi

Labels

None yet

Projects

None yet

Milestone

v0.18.0

Development

Successfully merging this pull request may close these issues.

3 participants

@Pouyanpi @codecov-commenter @tgasser-nv